On the dynamics of Vector Quantization and Neural Gas

A large variety of machine learning models which aim at vector quantization have been proposed. However, only very preliminary rigorous mathematical analysis concerning their learning behavior such as convergence speed, robustness with respect to initialization, etc. exists. In this paper, we use the theory of on-line learning for an exact mathematical description of the training dynamics of Vector Quantization mechanisms in model situations. We study update rules including the basic WinnerTakes-All mechanism and the Rank-Based update of the popular Neural Gas network. We investigate a model with three competing prototypes trained from a mixture of Gaussian clusters and compare performances in terms of dynamics, sensitivity to initial conditions and asymptotic results. We demonstrate that rank-based Neural Gas achieves both robustness to initial conditions and best asymptotic quantization error.